Interactive assistant

ABSTRACT

An interactive troubleshooting assistant and method for troubleshooting a system in real time to repair (fix) one or more problems in a system is disclosed. The interactive troubleshooting assistant and method may include receiving multimodal inputs from sensors, wearable devices, a person, etc. that may be input into a feature extractor including attention layers and pre-processing units of a cloud computing system hosted by one or more servers, such as a private cloud system. A pre-processing unit converts the raw multimodal input into a structed form so that an attention layer can give weights to features provided by the pre-processing unit according to their importance. The weighted extracted features may be provided to an actions predictor. The actions predictor generates the most suitable action based on the weighted extracted features generated by the feature extractor based on the multimodal inputs. After the most suitable action is performed, the interactive troubleshooting assistant considers new information from multimodal inputs so that the interactive troubleshooting assistant can provide the next recommended action. The interactive troubleshooting assistant may repeat these operations until the repair is completed.

CROSS-REFERENCE TO RELATED APPLICATION

This application is related to co-pending U.S. patent application Ser.No. 16/352,441, entitled “Interactive Troubleshooting Assistant”, filedon Mar. 13, 2019, which is hereby incorporated by reference in itsentirety.

TECHNICAL FIELD

The present disclosure relates to an interactive assistant. Morespecifically, the present disclosure generally relates to an interactiveassistant analyzing data received from multimodal inputs of smart andconnected components and/or a digital twin and a method for analyzingthe data. Even more specifically, the present disclosure generallyrelates to an interactive assistant for providing recommendations forfurther investigation and/or repair to a user after receiving at leastone of user input and data from smart and connected components and/or adigital twin to provide a next recommendation.

BACKGROUND

During manufacturing or operations, components, equipment, and processesof a system can fail partially or completely due to any number ofreasons and these components, equipment, and processes must be repairedto continue system operations. Troubleshooting is needed to examine thestate of a system and identify symptoms of one or problems (issues) withthe system so that an initial diagnosis or cause of the one or morefailures in a system can be determined. A field engineer or technicianmay be dispatched to troubleshoot the problem on the site of the systemand may attempt to follow a standard procedure to fix the problem, whichmay provide an initial recommended solution. The initial standardprocedure may include a sequence of instructions to take certainactions. However, the field engineer or technician may or may not havethe correct parts and tools to follow the standard procedure and to takethese actions. In addition, the initial diagnosis made by the fieldengineer or technician to fix the problem may or may not be accurate.Further, as the field engineer or technician is in the process ofrepairing or fixing the one or more problems based on the initialrecommended solution, the state of the system may change during thestandard procedure and therefore new information is generated. However,existing troubleshooting systems do not consider new information, whichis generated during the real time repair of components, equipment, orprocesses of a system and which is generated during inspection of thesystem by a field engineer or technician. Thus, the existingtroubleshooting systems provide a static diagnosis and static solution,which may include a static set of instructions or actions to be taken bya field engineer or technician. Accordingly, there is no method orapparatus to dynamically update the initial diagnosis and dynamicallychange the initial set of instructions or actions to be performed by thefield engineer or technician.

In addition, during installation of a new system, an installer such as afiled engineering or technician may follow a static set of installationinstructions for installation of a new system. Accordingly, there is nomethod or apparatus to dynamically update the installation instructionsand dynamically change the initial set of instructions or actions to beperformed by the field engineer or technician during installation.

There is a need in the art for a system and method that addresses theshortcomings of the prior art discussed above.

SUMMARY

An interactive troubleshooting assistant and method for troubleshootinga system is disclosed. The method includes troubleshooting a system inreal time to repair (fix) one or more problems (issues) by analyzingdata generated by smart and connected components and/or provided by auser, such as a technician or field engineer. The interactivetroubleshooting assistant and method solves at least the problemsdiscussed above by receiving information regarding the system andproviding an initial diagnosis of one or more problems in order toassign a correct one or more workers with the correct tools and parts(as needed) to repair the system, and by receiving and considering newinformation by monitoring the system during a repair process andadapting to new information after each action in the repair process isperformed to provide the next recommended action in the process forrepairing the system. The interactive troubleshooting assistant providesreal time recommendations for each action in the repair process byconsidering new information after each action is performed so that theinteractive troubleshooting assistant can provide the next recommendedaction. By modeling troubleshooting as a sequence to sequence predictionproblem and processing multimodal inputs, such as text, image, andsensor data from multiple sources, the interactive troubleshootingassistant and method considers new information after each action isperformed and recommends the next action until the system is repaired,so that the amount of time required to diagnose the one or more problemsand the amount of time required for repair of the system decreases.Accordingly, one or more embodiments utilize artificial intelligence toprovide a dynamically changing set of instructions so that actions canbe dynamically changed after completion of each action to decrease thetime to correct one or more system failures.

In an aspect, the disclosure provides a method of troubleshooting asystem for detecting a system failure and for dynamically recommendingone or more actions for repairing the detected failure of the system.The method may include receiving initial sensor inputs from internet ofthings (IOT) sensors and storing the initial sensor inputs as initialsensor data in a cloud server. The method may include analyzing theinitial sensor data to detect the system failure. The method may includedesignating a worker to repair the detected system failure based on theanalysis of the initial sensor data. The method may include repairingthe detected system failure by repeatedly performing operations untilthe failure is repaired. The operations may include receiving imagesfrom an imaging device and text from a mobile computing device andstoring the images as image data and the text as text data in the cloudserver. The operations may include receiving additional sensor inputsfrom internet of things (IOT) sensors and storing the additional sensorinputs as sensor data in the cloud server. The operations may includeextracting features from image data, text data, and sensor data,weighting the extracted features, and storing the weighted extractedfeatures in the cloud server. The operations may include recommending anaction based on the weighted extracted features and communicating theaction to a mobile computing device of the designated worker from thecloud server. The operations may include performing the action by thedesignated worker unless the recommended action from the cloud serverindicates that the system is repaired. The method may also includecommunicating, from the mobile computing device of the designated workerto the cloud server, a message which confirms that the system isrepaired.

In yet another aspect, the disclosure provides a non-transitory computerreadable medium storing software, which may include instructionsexecutable by one or more computers, which, upon such execution, causethe one or more computers to: (1) receive initial sensor inputs frominternet of things (IOT) sensors and store the initial sensor inputs asinitial sensor data in a cloud server; (2) analyze the initial sensordata to detect the system failure; (3) designate a worker to repair thedetected system failure based on the analysis of the initial sensordata; (4) repair the detected system failure by repeatedly performingoperations until the failure is repaired; and (5) communicate, from themobile computing device of the designated worker to the cloud server, amessage which confirms that the system is repaired. The operations mayinclude (1) receiving images from an imaging device and text from amobile computing device and storing the images as image data and thetext as text data in the cloud server; (2) receiving additional sensorinputs from internet of things (IOT) sensors and storing the additionalsensor inputs as sensor data in the cloud server; (3) extractingfeatures from image data, text data, and sensor data, weighting theextracted features, and storing the weighted extracted features in thecloud server; (4) recommending an action based on the weighted extractedfeatures and communicating the action to a mobile computing device ofthe designated worker from the cloud server; and (5) performing theaction by the designated worker unless the recommended action from thecloud server indicates that the system is repaired.

In yet another aspect, the disclosure provides an interactivetroubleshooting assistant for troubleshooting a system to detect asystem failure and for dynamically recommending one or more actions forrepairing the detected failure of the system, the interactivetroubleshooting assistant may include one or more computers and one ormore storage devices storing instructions that are operable, whenexecuted by one or more computers, to cause one or more computers to:(1) receive initial sensor inputs from internet of things (IOT) sensorsand store the initial sensor inputs as initial sensor data in a cloudserver; (2) analyze the initial sensor data to detect the systemfailure; (3) designate a worker to repair the detected system failurebased on the analysis of the initial sensor data; (4) repair thedetected system failure by repeatedly performing operations until thefailure is repaired; and (5) communicate, from the mobile computingdevice of the designated worker to the cloud server, a message whichconfirms that the system is repaired. The operations may include (1)receiving images from an imaging device and text from a mobile computingdevice and storing the images as image data and the text as text data inthe cloud server; (2) receiving additional sensor inputs from internetof things (IOT) sensors and storing the additional sensor inputs assensor data in the cloud server; (3) extracting features from imagedata, text data, and sensor data, weighting the extracted features, andstoring the weighted extracted features in the cloud server; (4)recommending an action based on the weighted extracted features andcommunicating the action to a mobile computing device of the designatedworker from the cloud server; and (5) performing the action by thedesignated worker unless the recommended action from the cloud serverindicates that the system is repaired.

In yet another aspect, the disclosure provides a method of installing asystem by dynamically recommending actions for system installation. Themethod may include accessing digital twin information from a digitaltwin stored on a cloud server through an interactive assistant stored onthe cloud server for system installation; assigning a designated workerby the interactive assistant for system installation; displaying aninstallation map on a display of the mobile communication device basedon digital twin information provided to the mobile communication devicethrough the interactive assistant; performing system installation byrepeatedly performing operations until the system is installed. Theoperations may include recommending action to be performed by thedesignated worker based on analysis of digital twin information andbased on images from an imaging device by using the interactiveassistant; performing the recommended action by the designated worker;determining whether recommended action is performed correctly byanalyzing the digital twin information and images from the imagingdevice using the interactive assistant; responding to an incorrectperformance of the recommended action by recommending the action to beperformed and performing the recommended action until the performance ofthe recommended action is correct using the interactive assistant;updating digital twin information when the recommended action isperformed correctly; and displaying updated map on the display of themobile computing device based on the updated twin information. After theinstalling is completed, the method may further include displaying thatthe installation is completed on the display of the mobile computingdevice may display that the installation is complete after the system isinstalled based on analysis of the interactive assistant.

In yet another aspect, the disclosure provides a non-transitory computerreadable medium storing software, which may include instructionsexecutable by one or more computers, which, upon such execution, causethe one or more computers to: (1) access digital twin information from adigital twin stored on a cloud server through an interactive assistantstored on the cloud server for system installation; (2) assign adesignated worker by the interactive assistant for system installation;(3) display an installation map on a display of a mobile communicationdevice based on digital twin information provided to the mobilecommunication device through the interactive assistant; and (4) performsystem installation by repeatedly performing operations until the systemis installed. The operations may include (1) recommending action to beperformed by the designated worker based on analysis of digital twininformation and based on images from an imaging device by using theinteractive assistant; (2) performing the recommended action by thedesignated worker; (3) determining whether recommended action isperformed correctly by analyzing the digital twin information and imagesfrom the imaging device using the interactive assistant; (4) respondingto an incorrect performance of the recommended action by recommendingthe action to be performed and performing the recommended action untilthe performance of the recommended action is correct using theinteractive assistant; (5) updating digital twin information when therecommended action is performed correctly; and (6) displaying updatedmap on the display of the mobile computing device based on the updatedtwin information. The method may further include displaying installationis completed on the display of the mobile computing device after thesystem is installed based on analysis of the interactive assistant. Theoperations may include

In yet another aspect, the disclosure provides an interactive assistantfor installing a system and for dynamically recommending one or moreactions for installing the system, the interactive assistant may includeone or more computers and one or more storage devices storinginstructions that are operable, when executed by one or more computers,to cause one or more computers to: (1) access digital twin informationfrom a digital twin stored on a cloud server through an interactiveassistant stored on the cloud server for system installation; (2) assigna designated worker by the interactive assistant for systeminstallation; (3) display an installation map on a display of a mobilecommunication device based on digital twin information provided to themobile communication device through the interactive assistant; (4)perform system installation by repeatedly performing operations untilthe system is installed. The operations may include (1) recommendingaction to be performed by the designated worker based on analysis ofdigital twin information and based on images from an imaging device byusing the interactive assistant; (2) performing the recommended actionby the designated worker; (3) determining whether recommended action isperformed correctly by analyzing the digital twin information and imagesfrom the imaging device using the interactive assistant; (4) respondingto an incorrect performance of the recommended action by recommendingthe action to be performed and performing the recommended action untilthe performance of the recommended action is correct using theinteractive assistant; (5) updating digital twin information when therecommended action is performed correctly; and (6) displaying updatedmap on the display of the mobile computing device based on the updatedtwin information. The method may further include displaying installationis completed on the display of the mobile computing device after thesystem is installed based on analysis of the interactive assistant.

Other systems, methods, features and advantages of the disclosure willbe, or will become, apparent to one of ordinary skill in the art uponexamination of the following figures and detailed description. It isintended that all such additional systems, methods, features andadvantages be included within this description and this summary, bewithin the scope of the disclosure, and be protected by the followingclaims.

While various embodiments are described, the description is intended tobe exemplary, rather than limiting and it will be apparent to those ofordinary skill in the art that many more embodiments and implementationsare possible that are within the scope of the embodiments. Although manypossible combinations of features are shown in the accompanying figuresand discussed in this detailed description, many other combinations ofthe disclosed features are possible. Any feature or element of anyembodiment may be used in combination with or substituted for any otherfeature or element in any other embodiment unless specificallyrestricted.

This disclosure includes and contemplates combinations with features andelements known to the average artisan in the art. The embodiments,features and elements that have been disclosed may also be combined withany conventional features or elements to form a distinct invention asdefined by the claims. Any feature or element of any embodiment may alsobe combined with features or elements from other inventions to formanother distinct invention as defined by the claims. Therefore, it willbe understood that any of the features shown and/or discussed in thepresent disclosure may be implemented singularly or in any suitablecombination. Accordingly, the embodiments are not to be restrictedexcept in light of the attached claims and their equivalents. Also,various modifications and changes may be made within the scope of theattached claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure can be better understood with reference to the followingdrawings and description. The components in the figures are notnecessarily to scale, emphasis instead being placed upon illustratingthe principles of the disclosure. Moreover, in the figures, likereference numerals designate corresponding parts throughout thedifferent views.

FIG. 1 is a block diagram of an embodiment of an interactivetroubleshooting assistant;

FIG. 2 shows an embodiment of a method of troubleshooting a system fordynamically recommending one or more actions for repairing the system;

FIG. 3 is a block diagram of an example of a paper mill system includingpaper machines for troubleshooting;

FIGS. 4A and 4B show an example of an embodiment of a method oftroubleshooting the paper mill machines in the paper mill system in FIG.3 ;

FIG. 5 is a diagram of an embodiment of an interactive troubleshootingassistant;

FIG. 6 is a diagram of an embodiment of a feature extractor of thetrouble shooting assistant in FIG. 5 ;

FIG. 7 is a diagram of an embodiment of an image preprocessor(preprocessing unit for images) of the feature extractor in FIG. 6 ;

FIG. 8 is a diagram showing an example of the application of convolutionand maxpooling of the image preprocessor of FIG. 7 ;

FIG. 9 is a diagram of an embodiment of a text preprocessor(preprocessing unit for text) of the feature extractor in FIG. 6 ;

FIG. 10 is a diagram showing an example of the application of embedding,convolution and maxpooling of the text preprocessor of FIG. 9 ;

FIG. 11 is a diagram of an embodiment of an IOT preprocessor(preprocessing unit for sensor data) of the feature extractor in FIG. 6;

FIG. 12 is a diagram showing an example of the application of thestatistical measures of the IOT preprocessor of FIG. 11 ;

FIG. 13 is a schematic diagram of an embodiment of a troubleshootingassistant applied to a paper mill machine;

FIG. 14 is a diagram of an embodiment of an actions predictor of thetrouble shooting assistant in FIG. 5 and FIG. 13 ;

FIG. 15 shows an embodiment of an actions predictor in FIG. 14 ;

FIG. 16 shows an embodiment of an actions predictor in FIG. 14 ;

FIG. 17 is a block diagram of an embodiment of an interactive assistant;

FIG. 18 shows an embodiment of a method of troubleshooting a system fordynamically recommending one or more actions for repairing the system;

FIG. 19 is a block diagram of an example of a paper mill systemincluding paper machines for troubleshooting;

FIGS. 20A, 20B, and 20C show an example of an embodiment of a method oftroubleshooting the paper mill machines in the paper mill system in FIG.19 ;

FIG. 21 is a diagram of an embodiment of an interactive assistant; and

FIG. 22 shows an embodiment of a method of installing a system bydynamically recommending one or more actions for installing the system.

DETAILED DESCRIPTION

An interactive troubleshooting assistant and method for troubleshootinga system is disclosed. The method includes troubleshooting a system inreal time to repair (fix) one or more problems (issues) by analyzingdata generated by smart and connected components and/or provided by auser, such as a technician or field engineer. Initially, an interactivetrouble shooting assistant analyzes data generated from smart andconnected components to provide a repair process and assign the correctdesignated worker or designated workers with the correct tools and partsto repair (fix) the system or a part of the system. The designatedworker or designated workers are hereinafter referred to as a designatedworker. The designated worker may be the user of the interactivetroubleshooting assistant. The designated worker may have one or moremobile (portable) computing devices, such as a smartphone, tablet,smartwatch, smart glass, or smart helmet. Although these mobilecomputing devices may be handheld or wearable, other mobile computingdevices, such as a laptop may be used. An application may be installedon the one or more mobile computing devices to communicate with theinteractive troubleshooting assistant to supply information to theinteractive troubleshooting assistant or to receive recommended actionsfrom the interactive troubleshooting assistant.

The interactive troubleshooting assistant enables the user tocomprehensively monitor the condition of one or more components of thesystem or the system itself by monitoring various operating parameters,environmental data, and other external data sources before and after thepartial failure or failure of one or more components in the system.Based on the evaluation of this data, the interactive troubleshootingassistant provides a detailed diagnosis of the problem (issue) and arecommended repair process along with a list of required tools andparts. The interactive troubleshooting assistant may receive newinformation from at least one of machine vision (image data), Internetof Things (IOT) sensor data and designated worker input (multimodalinputs) after each action in the repair process is performed by thedesignated worker. The IOT sensor data may be generated by IOT sensorsof smart and connected components of a system which includes one or moremachines. The interactive troubleshooting assistant adapts to the newinformation after each action in the repair process is performed toprovide the next recommended action in the repair process. Theinteractive troubleshooting assistant provides real time recommendationsfor each action in the repair process by considering new informationafter each action is performed so that the interactive troubleshootingassistant can provide the next recommended action. Once all the actionsare performed, the system is repaired.

For example, FIG. 1 is a block diagram of an embodiment of interactivetrouble shooting assistant 100 hosted by a cloud server 110 and coupledto a mobile computing device (portable electronic device) 120 andinternet of things (IOT) sensors 130 through the cloud server 110.Although the interactive trouble shooting assistant 100 hosted by acloud server 110 is shown coupled to one mobile computing device 120 inFIG. 1 , other embodiments may include one or more mobile computingdevices 120. The cloud server 110 may also be coupled wirelessly to oneor more mobile computing devices 120 and may be coupled wirelessly ordirectly to IOT sensors 130. The IOT sensors 130 provide IOTs sensordata (IOT sensor information) to the interactive trouble shootingassistant 100 hosted on the cloud server 110, so that the interactivetrouble shooting assistant 100 can monitor the state (condition) of asystem to identify or anticipate problems or issues which may cause orhave caused one or more failures in the monitored system. Although acloud server 110 is shown as an embodiment in FIG. 1 and other figures,any hosted server or a plurality of hosted servers may be used. One ormore cloud servers may be used. Any of these one or more cloud serversmay be public or private.

FIG. 2 shows an embodiment of a method of troubleshooting a system fordynamically recommending one or more actions for repairing the system.As discussed above, an interactive troubleshooting assistant 100monitors the state of a system and detects any issues (problems) thatare raised (operation 200). One or more issues (problems) are raised(detected) by an analysis of the IOT sensor data received by theinteractive trouble shooting assistant 100 (operation 200).

After at least one issue (problem) has been raised, the interactivetroubleshooting assistant 100 assigns a designated worker to inspect(investigate) the one or more issues (problems) (operation 210) raisedas a result of the monitoring by the interactive troubleshootingassistant 100 in operation 200. This assignment is based on one or moreissues being raised. In addition, as part of the designated workerassignment, the interactive trouble shooting assistant 100 may alsorecommend that one or more tools and/or one or more parts be taken bythe designated worker to the area (location) of the raised issue(operation 210). The interactive troubleshooting assistant 100 mayreceive multimodal inputs regarding the state of the system (operation220). These multimodal inputs may include the initial IOT sensor data,one or more images (image data), and text (text data). The image datamay be collected and provided to the interactive troubleshootingassistant 100 by an image capture device (e.g., a camera), which is partof the system, or by one or more image capture devices of one or moremobile computing devices 120. The captured image data may include one ormore images of a component of the system and/or an area around thecomponent of the system that the designated worker has been sent toinvestigate. The designated worker may possess one or more mobilecomputing devices 120. Examples of the one or more mobile computingdevices 120 include a smartphone, a tablet, a smartwatch, a smart glass,a smart helmet, and a laptop, which can communicate wirelessly with theinteractive troubleshooting assistant 100 hosted by the cloud server110. The designated worker may use the one or more mobile computingdevices 120 to input text which is wirelessly communicated to theinteractive troubleshooting assistant 100 hosted by the cloud server 110(operation 220). An application may be installed on the one or moremobile computing devices 120 to communicate with the interactivetroubleshooting assistant 100 to supply information to the interactivetroubleshooting assistant 100 or to receive recommended actions from theinteractive troubleshooting assistant 100.

Based on an analysis of the multimodal inputs, the interactivetroubleshooting assistant 100 recommends that the designated workerperform an action (operation 230). Examples of a designated workerinclude a technician, a field engineer, and a robot. The robot may be anautonomous robot. The designated worker may be a user. In thedisclosure, the designated worker may refer to one or more designatedworkers. In addition, if the designated worker is a robot, the mobilecomputing device 30 may be embedded in the robot (may be a part of therobot), may be held by the robot, or may be worn by the robot. Theseconfigurations are intended to be examples.

After the designated worker receives the recommended action through themobile computer device 120 from the interactive trouble shootingassistant 100 (operation 230), the designated worker performs therecommended action (operation 240). Thereafter, the interactivetroubleshooting assistant 100 receives additional data (additionalinformation) from one or more multimodal inputs regarding the state ofthe system after the recommended action is performed (operation 250).The interactive troubleshooting assistant 100 determines whether thesystem is repaired based on multimodal inputs received by theinteraction troubleshooting assistant 100 in operation 250 after therecommended action is performed (operation 260). These multimodal inputsmay include one or more of IOT sensor data, images (image data), or text(text data). When the interactive troubleshooting assistant 100determines that the system is repaired (one or more issues or problemsresolved) (operation 260), the interactive troubleshooting assistant 100recognizes the problem or issue has been repaired and continues tomonitor the system to detect any new issues or problems. When theinteractive troubleshooting assistant 100 determines that the system isnot repaired (operation 260), then the interactive troubleshootingassistant 100 returns to operation 230 so that another action can berecommended to the designated worker (operation 230) followed byoperations 240, 250, and 260 until the interactive troubleshootingassistant 100 determines that the system is repaired. The interactivetroubleshooting assistant 100 considers new information (new data) aftereach action is performed and recommends the next action until the systemis repaired, so that the amount of time required to diagnose the one ormore problems and the amount of time required for repair of the systemdecreases.

FIG. 3 is a block diagram of an example of a paper mill system 300including paper machine units (paper machines) 310-360, which can bemonitored by the interactive troubleshooting assistant 100 to detect anyissues (problems) caused by one or more failures in the paper millsystem. Each paper machine 310-360 has components.

FIGS. 4A and 4B show an example of an embodiment of a method oftroubleshooting the paper mill machines in the paper mill system in FIG.3 by using the interactive troubleshooting assistant 100 in FIG. 1 andthe interactive trouble shooting method shown in FIG. 2

In the example shown in FIGS. 4A and 4B, a failure is detected in papermachine unit 4 designated by reference numeral 340 of the paper millsystem 300 by the interactive troubleshooting assistant 100 shown inFIG. 1 (operation 400). Operation 400 in FIG. 4A generally correspondsto operation 200 (issue raised) in FIG. 2 . The IOT sensors 130 of FIG.1 provide IOTs sensor data (IOT sensor information) to the interactivetrouble shooting assistant 100 hosted on the cloud server 110, so thatthe interactive trouble shooting assistant 100 can monitor the state(condition) of a system to identify or anticipate problems or issueswhich may cause or have caused one or more failures in the monitoredsystem. One or more issues (problems) are raised (detected) by ananalysis of the IOT sensor data received by the interactive troubleshooting assistant 100 (operation 400). After the interactivetroubleshooting assistant 100 detects the failure in paper machine unit4, this exemplary method designates a worker (designated worker) torepair paper machine unit 4 (operation 405). Operation 405 in FIG. 4generally corresponds to operation 210 in FIG. 2 . The designated workermay be chosen based on one or more issues being raised. The interactivetroubleshooting assistant 100 may also recommend that one or more toolsand/or one or more parts be taken by the designated worker to the area(location) of the raised issue (operation 410). In addition, theinteractive troubleshooting assistant 100 may provide information to thedesignated worker by communicating with one or more mobile computingdevices 120 provided to the designated worker. An example of thisinformation is the conveyor belt in paper machine unit 4 has failed(operation 410). Operations 405 and 410 in FIG. 4 may be generallyperformed by operations 210 through 230 in FIG. 2 . However, operation230 in FIG. 2 can apply to other operations in FIG. 4 .

In this example, after receiving information from the interactivetrouble shooting assistant 100 through one or more mobile computingdevices 120, the designated worker visually inspects paper machine unit4 and collects image data through an image capture device of a wearabledevice (operation 415). A wearable device is an example of a mobilecomputing device 120. The image capture device provides one or moreimages of a significant leakage of fluid from machine unit 4 (operation420). In this example, the image data is wirelessly communicated to theinteractive trouble shooting assistant 100 in operation 420, and theimage data is validated by a text transmitted to the interactive troubleshooting assistant 100 by way of a mobile computing device 120(operation 425). The trouble shooting assistant 100 provides arecommendation to the designated worker based on the image data, thetext data, and IOT sensor data provided by IOT sensors 130. In operation430, the interactive troubleshooting assistant 100 recommends that thedesignated worker check for oil leakage from a sheet cutter component ofpaper machine unit 4 (operation 430). Operations 410-430 of FIG. 4 maybe generally performed by operations 220 and 230 in FIG. 2 .

The designated worker performs the recommended action of checking thesheet cutter or machine unit 4 for leakage and determines that the sheetcutter has not leaked. The designated worker also reports thisdetermination to the interactive troubleshooting assistant 100(operation 435). Operations 430 and 435 may be generally performed byapplying operations 240 through 260 of FIG. 2 . As indicated inoperation 440, another recommendation is made to the designated worker,which is similar to the operation 230 in FIG. 2 . More specifically, inoperation 440, the interactive troubleshooting assistant 100 providesthe designated worker with the recommendation to check a reducer of thepaper machine unit 4 for leakage. The designated worker performs theaction of inspecting the reducer and determines that the reducercomponent of paper machine unit 4 is leaking and the designated workerreports this determination to the interactive troubleshooting assistant100 (operation 445).

After the interactive troubleshooting assistant 100 determines that thereducer of the paper machine unit 4 is leaking fluid, thetroubleshooting assistant 100 provides the designated worker with arecommendation to apply adhesive and duct tape to the reducer to repairthe reducer (operation 450), and the designated worker applies adhesiveand duct tape to the reducer (operation 455). The interactivetroubleshooting assistant 100 also recommended that the designatedworker provide additional lubricants (fluids) to paper machine unit 4(operation 460), and the designated worker performed the action ofapplying lubricants to paper machine unit 4 (operation 465). Thereafter,the IOT sensors provide IOT sensor data to the interactivetroubleshooting assistant 100 including sensor data indicating thelubricant levels in paper machine unit 4 are satisfactory. Therefore,the interactive troubleshooting assistant 100 recommends a next actionto the designated worker, which is to start paper machine unit 4(operation 470). The designated worker starts paper machine unit 4 andreports the starting of the paper machine unit 4 to the interactivetroubleshooting assistant 100 (operation 475). The designated worker mayreport to the interactive troubleshooting assistant 100 by the mobilecomputing device 120. The designated worker may confirm that thedetected failure has been repaired and report this confirmation to theinteractive troubleshooting assistant 100 (operation 480).Alternatively, the designated worker and interactive troubleshootingassistant 100 may both confirm that the detected failure has beenrepaired (operation 480). Operation 480 is performed by applyingoperation 260 in FIG. 2 .

FIG. 5 is a diagram of an embodiment of a troubleshooting assistant 100.In an embodiment, a cloud server 110 hosts or includes the interactivetroubleshooting assistant 100 which is shown in FIG. 1 and FIG. 5 . Thetroubleshooting assistant 100 in FIG. 5 may include the featureextractor 510 of FIG. 5 , which converts multimodal inputs into astructured form which is suitable for an action predictor 520, whichrecommends a course of action to be performed. The interactivetroubleshooting assistant 100 may include the action predictor 520.Although a private cloud server is one embodiment, any privately hostedserver or a plurality of privately hosted servers may be used. One ormore private cloud servers may be used. In addition, one or more publiccloud servers may be used.

The feature extractor 510 includes a plurality of preprocessing units(preprocessors) 600, such as a preprocessing unit for images(preprocessor for images or image preprocessor) 610, a preprocessingunit for text (preprocessor for text or text preprocessor) 620, and apreprocessing unit for IOT sensor data (preprocessor for IOT sensor dataor IOT preprocessor) 630, which receive multimodal inputs which are dataor information from various devices. FIG. 6 is a diagram of anembodiment of a feature extractor of the trouble shooting assistant 100in FIG. 5 .

The image preprocessor 610 may receive image input from any type ofcamera or any other image capture device. An image capture device, suchas a camera may be installed as part of the monitored system or may bepart of a wearable device 120 worn by a designated worker, such as afield engineer or technician. The designated worker may also be a robot.The text preprocessor 620 may receive text by way of a mobile computingdevice 120. For example, the designated worker may input the text intothe mobile computing device 120 which transmits the inputted text to theinteractive troubleshooting assistant 100, so that the text preprocessorreceives the text inputted by the designated worker. In someembodiments, the text preprocessor 620 may receive a text that wasconverted from a user utterance (e.g., voice command) by aspeech-to-text system. The IOT preprocessor 630 receives sensor datafrom IOT sensors 130 which monitor the system. There may be one or morepreprocessing units 600 for IOT sensor data depending upon the type ofIOT sensor data. Multimodal inputs received by the respectivepreprocessing units are converted into a structured form and provideextracted relevant features from the multimodal inputs in the structuredform to respective attention layers 650. For example, images input to animage preprocessor 610 are converted into a suitable structured form,and extracted features in this suitable structured form are provided toan attention layer 660 for the extracted converted image input. Theattention layer 660 for images may provide weights to the extractedfeatures according to their importance. The action predictor 520, whichincludes deep neural network layers, may receive the weighted featuresto generate a recommended action to be taken by the designated worker.

One example of an image preprocessor 610 which receives input from theimage capture device may be a visual geometry group (VGG), such as VGG16(described by Karen Simonyan and Andrew Zisserman in “Very deepconvolutional networks for large-scale image recognition.” arXivpreprint arXiv:1409.1556 (2014)), which provides image classificationand feature extraction. Image pixels may be the image input supplied bythe image capture device to the image preprocessor of the featureextractor 510 of the interactive troubleshooting assistant 100. Sincethe image pixels may not be directly inputted into a deep learningmodel, an image preprocessor 610, such as the VGG16, can be used topreprocess the image pixels. FIG. 7 is a diagram of an embodiment of animage preprocessor (preprocessing unit for images) of the featureextractor in FIG. 6 . The image preprocessor 610 may correspond to theVGG16.

The VGG16 includes convolution layers and maxpooling layers. There is amaxpooling layer after every certain number of convolution layers. Morespecifically, if an image preprocessor 610 receives image pixels from animage capture device, these image pixels may be processed by a set ofconvolution layers. A convolution layer reduces a dimension of an inputby estimating the local relationships in the input. A maxpooling layerreceives the reduced dimension input and further reduces the dimensionof input by selecting the most important feature among the set offeatures provided by the corresponding set of convolution layers. Inexemplary embodiments, there may be one or more maxpooling layers whereeach maxpooling layer has a corresponding set of convolution layers. Aset of convolution layers may be one or more convolution layers. Amaxpooling layer provides extracted features to an image attention layer660 which assigns weights to the extracted features and which suppliesthe weighted extracted features to the actions predictor 520.

FIG. 8 is a diagram showing an example of the application of convolutionand maxpooling of the image preprocessor of FIG. 7 . More specifically,image pixels representing an image of paper machine unit 4 are shown ina six by six matrix. As indicated above, one or more convolution layersreduces a dimension of an input by estimating the local relationships inthe input. For example, pixel values 0, 1, 6, and 7 in the upperleft-hand side of the six by six matrix form a two by two matrix. Thesepixel values of the two by two matrix are added and the sum of thesepixel values is 14. The sum 14 is shown in the upper righthand corner inthe three by three matrix, which represents the local relation of pixelsof the two by two matrix in the six by six matrix. Similarly, pixelvalues 2, 3, 8, and 9 of the six by six matrix are added and the sum ofthese pixel values is 22 as shown in the three by three matrix. In thisexample, the six by six matrix is divided into a group of 2 by 2matrices so that local relationships among pixels in the six by sixmatrix are determined by the one or more convolution layers to provide athree by three matrix. Thereafter, a maxpooling layer is applied to thethree by three matrix to provide a two by two matrix. In this example,the maxpooling layer processes the three by three matrix by determiningthe maximum pixel value among two by two matrices in the three by threematrix in the following manner. For example, the maximum number of afirst set of pixel values 14, 22, 62, and 70 is 70, which appears at theupper left-hand corner of the two by two matrix formed after themaxpooling layer has been applied to the three by three pixel matrix.The maximum number for a second set of pixel values 62, 70, 110, and 118is 118 as shown in the lower left-hand corner of the two by two matrixformed after the maxpooling layer has been applied to the three by threepixel matrix. The maximum number for a third set of pixel values 22, 30,70, and 78 is 78 as shown in the upper right-hand corner of the two bytwo matrix formed after the maxpooling layer has been applied to thethree by three pixel matrix. The maximum number for a fourth set ofpixel values 70, 78, 118, and 126 is 126 as shown in the lowerright-hand corner of the two by two matrix formed after the maxpoolinglayer has been applied to the three by three pixel matrix. The pixelvalues of the extracted two by two matrix represent one or moreextracted features, which are forwarded to the image attention layer 660of FIG. 6 for further processing in the feature extractor 510.

In another example, a designated worker may input text into a mobilecomputing device 120 which transmits the input text to the textpreprocessor 620 of the feature extractor 510 of the interactivetroubleshooting assistant 100. Since text cannot be transmitted directlyto a deep learning module, a text preprocessor 620 can be used topreprocess the text. FIG. 9 is a diagram of an embodiment of a textpreprocessor (preprocessing unit for text) of the feature extractor inFIG. 6 . As shown in FIG. 9 , one example of a text preprocessor 620,which may receive input from a mobile computing device and may outputextracted features to an image attention layer 670, may include anembedding layer, a convolution layer, and a maxpooling layer. Anembedding layer may convert input text to a word embedding, which are amodel for converting input text into one or more mathematical vectors.More specifically, the embedding layer converts words into vectorrepresentations. The embedding layer may utilize word embeddings whichare semantic vector space representations that represent each word witha meaningful vector. Examples of a word embedding include a gloveembedding and a word2vec embedding. These representations are receivedby a convolution layer to reduce the dimension of the input byestimating the local relationships in the input. The maxpooling layerreceives the reduced dimension input from the convolution layer tofurther reduce the dimension of the input by selecting the mostimportant feature among the set of features. The maxpooling layerprovides extracted features to the text attention layer 670 whichassigns weights to the extracted features and which supplies theweighted extracted features to the actions predictor 520.

FIG. 10 is a diagram showing an example of the application of embedding,convolution, and maxpooling of the text preprocessor 620 of FIG. 9 . Inthis example, word embedding is used to provide vectors corresponding towords, which are then placed in a matrix. In this example, the matrix isa three by three matrix. As indicated above, these representations ofvectors are received by a convolution layer to reduce the dimension ofthe input by estimating the local relationships in the input. Forexample, the values 0.1, 0.2, 0.4, and 0.5 are added and the sum ofthese values is 1.2, which appears in the upper left-hand corner of thetwo by two matrix in FIG. 10 . After the application of the convolutionlayer, the values 0.4, 0.7, 0.5, and 0.8 are added and the sum 2.4appears in the lower left-hand corner of the two by two matrix of FIG.10 . After the application of the convolution layer, the values 0.2,0.3, 0.5, and 0.6 are added and the sum 1.6 appears in the upperright-hand corner of the two by two matrix of FIG. 10 . After theapplication of the convolution layer, the values 0.5, 0.6, 0.8, and 0.9are added and the sum 2.8 appears in the lower right-hand corner of thetwo by two matrix of FIG. 10 . After the convolution layer in the threeby three matrix shown in FIG. 10 is applied to produce a two by twomatrix shown in FIG. 10 , a max pooling layer is applied to the two bytwo matrix in FIG. 10 and the maximum value represents the one or moreextracted features of the text data, which are forwarded to the textattention layer 670 as shown in FIG. 6 .

In another example, the interactive troubleshooting assistant 100 mayreceive sensor data (sensor information) from one or more IOT sensors130 monitoring the system. Since sensor data cannot be transmitteddirectly to a deep learning module, an IOT preprocessor 630 can be usedto preprocess sensor data. FIG. 11 is a diagram of an embodiment of anIOT preprocessor (preprocessing unit for sensor data) 630 of the featureextractor 510 in FIG. 6 . One example of an IOT preprocessor 630, whichmay receive input from one or more IOT sensors 130 and may outputextracted features to an IOT attention layer 680, may include astatistical measure and a dense layer for each IOT sensor 130. Thestatistical measures may include one or more of an arithmetic mean, amedian, a standard deviation, a standardized moment, a kurtosis, askewness, an entropy, and a root mean squared. The statistical measuresfor an IOT sensor 130 are passed to a dense layer corresponding to theIOT sensor 130. For example, the eight statistical measures of an IOTinput from a first IOT sensor 130 may be represented by a tuple (x₁, x₂,. . . , x₈). However, this is an example and there may be more or lessthan eight statistical measures. After passing through the dense layer,these statistical measures (x₁, x₂, . . . , x₈) are mapped to a valuethat is some combination of all these statistical measures, i.e.y_(i)=f(x₁, x₂, . . . , x₈). A mapped value, representing thecombination of all statistical measures, for each IOT input from eachIOT sensor 130, can be represented by another tuple (y₁, y₂, . . . ,y_(n)) where n represents the number of sensors. This tuple (y₁, y₂, . .. , y_(n)) may be passed to another dense layer that maps the tuple (y₁,y₂, . . . , y_(n)) to a value which is some combination of all of thesey_(i) (1≤I≤n), which may be represented by z=f(y₁, y₂, . . . , y_(n)).The another dense layer provides extracted features to the IOT attentionlayer 680 which assigns weights to the extracted features and whichsupplies the weighted extracted features to the actions predictor 520.

FIG. 12 is a diagram showing an example of the application of thestatistical measures of the IOT preprocessor 630 of FIG. 11 . FIG. 12shows a pressure vector, a temperature vector, and a humidity vector,which are based on IOT sensor data. The IOT sensor data is thenprocessed to provide statistical measures of the IOT sensor data to beprocessed by one or more dense layers. FIG. 13 shows the extractedfeatures from the one or more dense layers (Pressure 0.3212, Temperature0.1192, and Humidity −0.2674).

After features are extracted by the image preprocessor 610, textpreprocessor 620, and IOT preprocessor 630, an attention layer 650 asshown in FIG. 6 may utilize an attention model to give importance todesired features (input features) of a multimodal input, which areextracted by using an image preprocessor 610, a text preprocessor 620,and/or an IOT preprocessor 630. As indicated above, the imagepreprocessor 610 may have a corresponding image attention layer 660, thetext preprocessor 620 may have a corresponding text attention layer 670,and the IOT preprocessor 630 may have a corresponding IOT attentionlayer 680 to provide weights to extracted features. Generally, theextracted features may be multiplied with corresponding attentionprobabilities to give weighted features. An example of an attentionmodel can be illustrated by the following mathematical equation:(b ₁ ,b _(2,) . . . ,b _(n))(a ₁ ,a ₂ , . . . ,a _(n))=(x ₁ a ₁ ,x ₂ a ₂, . . . x _(n) a _(n))

The variables b₁, b₂, . . . , b_(n) constitute extracted features andthe variables a₁, a₂, . . . a_(n) constitute attention probabilities.The attention probability a_(i) indicates the significance of extractedfeatures b_(i) or the amount of attention that needs to be given toextracted feature b₁.

In an exemplary embodiment, the features, which had been extracted andweighted by the feature extractor 510, are passed to the actionpredictor 520 to generate a recommended action for the designated workeras shown in FIG. 5 . These features passed from the feature extractor510 to the action predictor 520 are referred to as extracted features.Based on these extracted features, the actions predictor 520 cangenerate the most suitable action.

FIG. 13 is a schematic diagram of an embodiment of a troubleshootingassistant 100 applied to a paper mill machine unit 4 designated byreference numeral 340, which shows the results of the image preprocessor610, text preprocessor 620, and IOT preprocessor 130 operating on datain the examples shown in FIG. 8 , FIG. 10 , and FIG. 12 . The mostimportant features are concatenated as shown in FIG. 13 and attentionprobabilities are applied to the most important features to provide theappropriate weights to the most important features. As shown in FIG. 13, the actions predictor 520 predicts a course of action based on theinformation supplied by the feature extractor 510 and may also considerpast input states (past image data, text data, and sensor data) as well.

As indicated above, actions predictor 520 is shown in FIG. 5 and FIG. 13. FIG. 14 is a is a diagram of an embodiment of an actions predictor 520of the trouble shooting assistant 100 in FIG. 5 and FIG. 13 . The actionpredictor 520 may include a long short term memory (LSTM) which is adeep neural network. An LSTM has the capability to bridge long time lagsbetween inputs and can remember inputs from the past for a longduration. This capability makes the LSTM advantageous for learning longsequences.

In an embodiment, n LTSM layers may be stacked on top of each other.FIG. 15 shows an embodiment of an actions predictor in FIG. 14 . Attimestamp t, the input from the previous LSTM layer as well as theprevious state of the current LSTM layer at timestamp t−1 are passed tothe current LSTM layer. This ensures that for any problem (issue) whichhas been raised, the state of the system reflected by the current inputas well as the state of the system reflected by previous inputs (e.g.all previous inputs) is considered while predicting new actions. Thestate of the system may only be a state of part of the system as well. Adense layer is added on top of the LSTM layer to convert the output ofthe LSTM Layer n to the desired form for the designated worker.

As shown in FIG. 16 which is applicable to the paper mill machine unit 4of the paper mill system 300, an example of the handling of a text inputis shown. If the designated worker inputs the following text “Papermachine has stopped working and no production of paper is happening inthe Unit 4” at timestamp 1. The input after passing through the featureextractor 510 passes through the LSTM layers. Each LTSM layer maintainsa state S_(1,1) that represents this input. The last action orrecommendation predicted by the actions predictor 520 is “Check ReducerComponent.” The designated worker may give the next input “The ReducerComponent Looks Fine” at timestamp 2. When this new input passes throughthe LTSM layers, the state S_(1,1) as well as the new worker input isconsidered to predict the new state S_(1,12).

The interactive troubleshooting assistant 100 outputs the recommendedaction, provided by the action predictor 520, to one or more mobilecomputing devices 120 so the designated workers receive the recommendedaction. As discussed above, once the recommended action is performed,the status of the system after the action is taken is evaluated by theinteractive troubleshooting assistant 100, which analyzes the newinformation (new data) from new images, new text input, and/or new IOTsensor data generated after the action is performed, and the interactivetroubleshooting assistant 100 provides the next recommended action. Thisprocess may be repeated until the repair is complete. By dynamicallychanging the recommended action until the repair to the system iscompleted, the overall repair time may be greatly reduced.

Although the above embodiments have referred to dispatching one or morefield engineering, technician, or robots (designated worker), thedesignated worker may also be any other machine, and the user of theinteractive troubleshooting assistant 100 may a robot or machine whichuses or includes a mobile computing device 120, which has an applicationfor communication with the interactive troubleshooting assistant 100.

In another example, FIG. 17 shows a block diagram of an embodiment of aninteractive assistant 1710 hosted by a cloud server 1700 and coupled toa mobile computing device (portable electronic device) 1730 and internetof things (IOT) sensors 1740 through the cloud server 1700. Theinteractive assistant 1710 may be coupled to a digital twin 1720, whichis also hosted by the cloud server 1700. A digital twin 1720 is adigital representation of one or more physical objects. A digital twin1720 may provide a virtual replica of physical objects such asbuildings, machines, components, robots, and persons. The digital twin1720 may also provide identification information of physical objects aswell as the physical characteristics, properties, and or uses of thephysical objects. In addition, the digital twin 1720 may provide ahistory of the use of a physical objection. All of this informationprovided by digital twin 1720 may be referred to as digital twininformation.

Although the interactive assistant 1710 hosted by a cloud server 1700 isshown coupled to one mobile computing device 1730 in FIG. 1 , otherembodiments may include one or more mobile computing devices 1730. Thecloud server 1700 may also be coupled wirelessly to one or more mobilecomputing devices 1730 and may be coupled wirelessly or directly to IOTsensors 1730. Along with the digital twin information provided bydigital twin 1720 to the interactive assistant 1710, the IOT sensors1730 provide IOTs sensor data (IOT sensor information) to theinteractive assistant 1710 hosted on the cloud server 1700, so that theinteractive assistant 1710 can monitor the state (condition) of a systemto identify or anticipate problems or issues which may cause or havecaused one or more failures in the monitored system. Although a cloudserver 1700 is shown as an embodiment in FIG. 17 and other figures, anyhosted server or a plurality of hosted servers may be used. One or morecloud servers may be used. Any of these one or more cloud servers may bepublic or private.

FIG. 18 shows an embodiment of a method of troubleshooting a system fordynamically recommending one or more actions for repairing the system byaccessing a digital twin 1720 in addition to other information such asthe sensor data supplied by the internet of things sensors 1730. Asdiscussed above, the interactive assistant 1710 monitors the state of asystem and detects any issues (problems) that are raised (operation1810). One or more issues (problems) are raised (detected) by ananalysis of the IOT sensor data which is received by the interactiveassistant 1710 (operation 1810). The digital twin 1720 is notified ofthe issue raised (operation 1820) and the digital twin may communicatedigital twin information to the interactive assistant (operation 1830).Alternatively, once the issues is raised (operation 1810), theinteractive assistant 1710 accesses the digital twin 1720 to requestdigital twin information.

After at least one issue (problem) has been raised and the digital twin1720 has communicated with the interactive assistant 1710, theinteractive assistant 1710 assigns a designated worker to inspect(investigate) the one or more issues (problems) (operation 1840) raisedas a result of the monitoring by the interactive assistant 1710 inoperation 200. This assignment is based on one or more issues beingraised as well as digital twin information provided by the digital twin1720. In addition, as part of the designated worker assignment, theinteractive assistant 1710 may also recommend that one or more toolsand/or one or more parts be taken by the designated worker to the area(location) of the raised issue (operation 1840). The interactiveassistant 1710 may receive multimodal inputs regarding the state of thesystem (operation 1850). These multimodal inputs may include the initialIOT sensor data, one or more images (image data), and text (text data).The image data may be collected and provided to the interactiveassistant 1710 by an image capture device (e.g., a camera), which ispart of the system, or by one or more image capture devices of one ormore mobile computing devices 1730. The captured image data may includeone or more images of a component of the system and/or an area aroundthe component of the system that the designated worker has been sent toinvestigate. The designated worker may possess one or more mobilecomputing devices 1730. Examples of the one or more mobile computingdevices 1730 include a smartphone, a tablet, a smartwatch, a smartglass, a smart helmet, and a laptop, which can communicate wirelesslywith the interactive assistant 1710 hosted by the cloud server 1700. Thedesignated worker may use the one or more mobile computing devices 1730to input text which is wirelessly communicated to the interactiveassistant 1710 hosted by the cloud server 1700 (operation 1850). Anapplication may be installed on the one or more mobile computing devices1730 to communicate with the interactive troubleshooting assistant 1710to supply information to the interactive troubleshooting assistant 1710or to receive recommended actions from the interactive troubleshootingassistant 1710.

Based on an analysis of the multimodal inputs and the digital twininformation, the interactive assistant 1710 recommends that thedesignated worker perform an action (operation 1860). Examples of adesignated worker include a technician, a field engineer, and a robot.The robot may be an autonomous robot. The designated worker may be auser. In the disclosure, the designated worker may refer to one or moredesignated workers. In addition, if the designated worker is a robot,the mobile computing device 1730 may be embedded in the robot (may be apart of the robot), may be held by the robot, or may be worn by therobot. These configurations are intended to be examples.

After the designated worker receives the recommended action through themobile computer device 1730 from the interactive assistant 1710(operation 1860), the designated worker performs the recommended action(operation 1870) and the digital twin information in the digital twin1720 is updated (operation 1880). Thereafter, the interactive assistant1710 receives additional data (additional information) from one or moremultimodal inputs regarding the state of the system after therecommended action is performed. The interactive troubleshootingassistant 1710 determines whether the system is repaired based onmultimodal inputs received by the interaction assistant 1710 and theupdated digital twin information after the recommended action isperformed (operation 1890). These multimodal inputs may include one ormore of IOT sensor data, images (image data), or text (text data). Whenthe interactive assistant 1710 determines that the system is repaired(one or more issues or problems resolved) (operation 1890), theinteractive assistant 1710 recognizes the problem or issue has beenrepaired and continues to monitor the system to detect any new issues orproblems. When the interactive assistant 1710 determines that the systemis not repaired (operation 1890), then the interactive assistant 1710returns to operation 1860 so that another action can be recommended tothe designated worker (operation 1860) followed by operations 1870,1880, and 1890 until the interactive assistant 1710 determines that thesystem is repaired. The interactive assistant 1710 considers newinformation (new data) including updated digital twin information andmultimodal inputs after each action is performed and recommends the nextaction until the system is repaired, so that the amount of time requiredto diagnose the one or more problems and the amount of time required forrepair of the system decreases.

FIG. 19 is a block diagram of an example of a paper mill system 1900including paper machine units (paper machines) 1910-1960, which can bemonitored by the interactive assistant 1710 to detect any issues(problems) caused by one or more failures in the paper mill system. Eachpaper machine 1910-1860 has components. In this example, the digitaltwin 1720 may store digital twin information of the paper mill system1900 which include digital twin information regarding paper machineunits 1910-1960 any components related to the paper machine units1910-1960. The cloud server 1700 hosts the digital twin 1720 whichincludes the digital twin information of paper mill system 1900.

FIGS. 20A, 20B, and 20C show an example of an embodiment of a method oftroubleshooting the paper mill machines in the paper mill system in FIG.19 by using the interactive assistant 1710 in FIG. 17 and theinteractive trouble shooting method shown in FIG. 18 .

In the example shown in FIGS. 20A, 20B, and 20C, a failure is detectedin paper machine unit 4 designated by reference numeral 1940 of thepaper mill system 1900 by the interactive assistant 1710 shown in FIG.17 (operation 2000). Operation 2000 in FIG. 20A generally corresponds tooperation 1810 (issue raised) in FIG. 18 . The IOT sensors 1740 of FIG.17 provide IOTs sensor data (IOT sensor information) to the interactiveassistant 1710 hosted on the cloud server 1700, so that the interactiveassistant 1710 can monitor the state (condition) of a system to identifyor anticipate problems or issues which may cause or have caused one ormore failures in the monitored system. One or more issues (problems) areraised (detected) by an analysis of the IOT sensor data received by theinteractive assistant 1710 (operation 2000). After the interactiveassistant 1710 detects the failure in paper machine unit 4, thisexemplary method notifies a digital twin that an issue has been raised(operation 2002). Operation 2002 generally corresponds to operation 1820in FIG. 18 . The digital twin 1720 is notified of the issue raised(operation 2002) and the digital twin may communicate digital twininformation to the interactive assistant (operation 2004). Operation2004 generally corresponds to operation 1830 in FIG. 18 . Alternatively,once the issues is raised (operation 2000), the interactive assistant1710 accesses the digital twin 1720 to request digital twin information.

The interactive assistant 1710 designates a worker (designated worker)to repair paper machine unit 4 (operation 2006). Operation 2006 in FIG.20A generally corresponds to operation 1840 in FIG. 18 . This assignmentis based on one or more issues being raised as well as digital twininformation provided by the digital twin 1720. In addition, as part ofthe designated worker assignment, the interactive assistant 1710 mayalso recommend that one or more tools and/or one or more parts be takenby the designated worker to the area (location) of the raised issue(operation 2006).

In addition, the interactive assistant 1710 may provide information tothe designated worker by communicating with one or more mobile computingdevices 1730 provided to the designated worker. An example of thisinformation is the conveyor belt in paper machine unit 4 has failed(operation 2008). Operations 2006 and 2008 in FIG. 20A may be generallyperformed by operations 1830 through 1860 in FIG. 18 . However,operation 1860 in FIG. 18 can apply to other operations in FIGS.20A-20C.

In this example, after receiving information from the interactiveassistant 1710 through one or more mobile computing devices 1730, thedesignated worker visually inspects paper machine unit 4 and collectsimage data through an image capture device of a wearable device(operation 2010). A wearable device is an example of a mobile computingdevice 1730. The image capture device provides one or more images of asignificant leakage of fluid from machine unit 4 (operation 2012). Inthis example, the image data is wirelessly communicated to theinteractive assistant 1710 in operation 2012, and the image data isvalidated by a text transmitted to the interactive assistant 1710 by wayof a mobile computing device 120 (operation 2014). The interactiveassistant 1710 provides a recommendation to the designated worker basedon the image data, the text data, and IOT sensor data provided by IOTsensors 1740. In operation 2016, the interactive assistant 1710recommends that the designated worker check for oil leakage from a sheetcutter component of paper machine unit 4 (operation 2016). Operations2008-2016 of FIG. 20A may be generally performed by operations 1850 and1860 in FIG. 18 .

The designated worker performs the recommended action of checking thesheet cutter or machine unit 4 for leakage and determines that the sheetcutter has not leaked. The designated worker also reports thisdetermination to the interactive assistant 1710 (operation 2018 in FIG.20B). Operations 2016 and 2018 may be generally performed by applyingoperations 1870 through 1890 of FIG. 18 . As indicated in operation2020, another recommendation is made to the designated worker, which issimilar to the operation 1860 in FIG. 18 . More specifically, inoperation 2020, the interactive assistant 1710 provides the designatedworker with the recommendation to check a reducer of the paper machineunit 4 for leakage. This recommendation may be based upon an indicationfrom digital twin 1720 that the reducer is nearing the end ofrecommended usage (e.g. 2,000 hours of use). If the digital twin 1720indicated that the reducer was new, a different recommendation by theinteractive assistant 1710 may have been made. The designated workerperforms the action of inspecting the reducer and determines that thereducer component of paper machine unit 4 is leaking and the designatedworker reports this determination to the interactive assistant 1710(operation 2022).

After the interactive assistant 1710 determines that the reducer of thepaper machine unit 4 is leaking fluid, the interactive assistant 1710provides the designated worker with a recommendation to replace thereducer with a spare located in an inventory cage 2 (operation 2024) inthis example. The designated worker replaces the reducer, thereplacement of the reducer is reported to interactive assistant 1710,and the serial number is reported to the digital twin 1720 to reset thetracking data in the digital twin through the interactive assistant 1710(operation 2026). The interactive assistant 1710 also recommended thatthe designated worker provide additional lubricants (fluids) to papermachine unit 4 (operation 2028), and the designated worker performed theaction of applying lubricants to paper machine unit 4 (operation 2030).Thereafter, the IOT sensors 1740 provide IOT sensor data to theinteractive assistant 1710 including sensor data indicating thelubricant levels in paper machine unit 4 are satisfactory. Therefore,the interactive assistant 1710 recommends a next action to thedesignated worker, which is to start paper machine unit 4 (operation2032 in FIG. 20C). The digital twin information in digital twin 1720 isupdated through one or more communications from the interactiveassistant 1710 to provide any additional updates to the digital twin1720 (operation 2034). The designated worker starts paper machine unit 4and reports the starting of the paper machine unit 4 to the interactiveassistant 1710 (operation 2036). The designated worker may report to theinteractive assistant 1710 by the mobile computing device 1730. Thedesignated worker may confirm that the detected failure has beenrepaired and report this confirmation to the interactive assistant 1710(operation 2038). Alternatively, the designated worker and interactiveassistant 1710 may both confirm that the detected failure has beenrepaired (operation 4038).

FIG. 21 is a diagram of an embodiment of an interactive assistant 1710which is configured to access past input states 2130 and digital twininformation 2140. In an embodiment, a cloud server 1700 hosts orincludes the interactive assistant 1710. The interactive assistant 1710in FIG. 21 may include the feature extractor 2100 which is similar tothe feature extractor 510 of FIG. 5 , which converts multimodal inputsinto a structured form which is suitable for an action predictor 2120,which recommends a course of action to be performed. Placing themultimodal inputs into the structure form includes applying weights toextracted features as denoted by reference numeral 2110. The actionpredictor 2120 also receives digital twin information 2140 and pastinput states of multimodal inputs in order to recommend a course ofaction. The actions predictor 2120 may predict a course of action basedon the information supplied by the feature extractor 2100, past inputstates (past image data, text data, and sensor data) from the multimodalinputs, and digital twin information. Although a private cloud server isone embodiment which hosts the digital twin 1720 and interactiveassistant 1710, any privately hosted server or a plurality of privatelyhosted servers may be used. One or more private cloud servers may beused. In addition, one or more public cloud servers may be used.

FIG. 22 shows an embodiment of a method of installing a system bydynamically recommending one or more actions for installing the system.In order to install a system, the interactive assistant 1710 accessesthe digital twin 1720 to obtain digital twin information (operation2205). As discussed above, a digital twin 1720 may provide a virtualreplica of physical objects such as buildings, machines, components,robots, and persons. The digital twin 1720 may also provideidentification information of physical objects as well as the physicalcharacteristics, properties, and or uses of the physical objects. Inaddition, the digital twin 1720 may provide a history of the use of aphysical objection. All of this information provided by digital twin1720 may be referred to as digital twin information.

The digital twin 1720 communicates digital twin information includinginstallation information to the interactive assistant 1710 (operation2210). Based on the digital twin information, a designated worker isassigned by the interactive assistant (operation 2215). The designatedworker has a mobile computing device 1730 which may have an applicationwhich allows the computing device 1730 to communicate with theinteractive assistant 1710. As a result of this communication, theinteractive assistant 1710 communicates with the mobile computing device1730 to display on a display of the mobile computing device 1730 a mapbased on the digital twin information supplied by the digital twin 1720through the interactive assistant 1710 (operation 2220).

The interactive assistant recommends an action to be performed based onan analysis of the digital twin information (operation 2225). Therecommended action may be communicated to the designated worker throughthe mobile computing device 1730 using a display to display therecommended action (operation 2225). The mobile computing device 1730may also have a speaker so that the recommended action may becommunicated through a speaker (operation 2225). The designated workerperforms the recommended action (operation 2230). If the recommendedaction is not performed correctly (operation 2235), the method mayreturn to operation 2325 so that the designated worker receives arecommended action. If the recommended action is performed correctly,the interactive assistant 1710 communicates with the digital twin 1720to update the digital twin information (operation 2240). The interactiveassistant 1710 communications with the mobile computing device 1730 todisplay an updated map on the mobile communication device 1730 based onthe updated digital twin information (operation 2245). If theinstallation is completed (operation 2250), then the installation ends.However, if the installation is incomplete, the method returns tooperation 2225 for the next recommended action to perform to install thesystem until the system is installed.

Although the above embodiments have referred to dispatching one or morefield engineering, technician, or robots (designated worker), thedesignated worker may also be any other machine, and the user of theinteractive assistant 1710 may a robot or machine which uses or includesa mobile computing device 1730, which has an application forcommunication with the interactive assistant 1710 which can access thedigital twin 1720.

While various embodiments have been described above, the description isintended to be exemplary, rather than limiting and it will be apparentto those of ordinary skill in the art that many more embodiments andimplementations are possible that are within the scope of the invention.Accordingly, the invention is not to be restricted except in light ofthe attached claims and their equivalents. Also, various modificationsand changes may be made within the scope of the attached claims.

We claim:
 1. A method of installing a system by dynamically recommendingactions for system installation, the method comprising: accessinginitial digital twin information from a digital twin stored on a cloudserver through an interactive assistant stored on the cloud server forsystem installation; analyzing the initial digital twin information todetect a system failure; assigning a designated worker by theinteractive assistant for system installation based on the analysis ofthe initial digital twin information, wherein the designated worker is arobot and a mobile computing device is embedded in the robot; displayingan installation map on a display of the mobile computing device based onthe initial digital twin information provided to the mobile computingdevice through the interactive assistant; performing system installationby repeatedly performing operations until determining that the systemfailure is repaired, the operations comprising: receiving images from animaging device and text from the mobile computing device and storing theimages as image data and the text as text data in the cloud server,wherein the images show one or more components of the system beinginstalled and the text includes input from the designated worker;receiving updated digital twin information; determining whether thefailure is repaired based on the image data, text data, and updateddigital twin data; upon determining that the failure is not repaired,determining a first action to be performed by the designated workerbased on analysis of the image data, text data, and the initial digitaltwin information, wherein the first action includes checking for anissue with a first component; upon receiving a communication from thedesignated worker that the issue does not exist with the firstcomponent, recommending that the designated worker check for the issuewith a second component; upon receiving a communication from thedesignated worker that the issue exists with the second component,determining a second action to be performed by the designated workerbased on analysis of the image data, text data, and the initial digitaltwin information; and recommending the second action to the designatedworker via communication to the mobile computing device from the cloudserver; and displaying that the failure is repaired on the display ofthe mobile computing device after the system is installed based onanalysis of the interactive assistant.
 2. The method of claim 1, whereindetermining the second action includes: converting the image data, textdata, and updated digital twin data into a structured form, extractingfeatures from the structured form, and applying weights to extractedfeatures to create weighted features; applying an action predictorincluding deep neural network layers to the weighted features togenerate the second action.
 3. The method of claim 1, wherein theinitial digital twin information comprises a virtual representation ofthe system installed in a building or a virtual representation of acomponent for the system.
 4. The method of claim 1, wherein the issue isan oil leak.
 5. The method of claim 1, further comprising recommending atool for the designated worker to use based on analysis of the imagedata, text data, and the initial digital twin information.
 6. The methodof claim 1, wherein the operation of recommending the second action tobe performed includes outputting the recommended second action to thedesignated worker by using the display and a speaker of the mobilecomputing device.
 7. The method of claim 1, further comprising analyzingthe initial digital twin information to determine that a component ofthe system being installed is nearing end of recommended usage andwherein the second action recommended includes replacing the component.8. A non-transitory computer readable medium storing software comprisinginstructions executable by one or more computers, which, upon suchexecution, cause the one or more computers to: access initial digitaltwin information from a digital twin stored on a cloud server through aninteractive assistant stored on the cloud server for systeminstallation; analyze the initial digital twin information to detect asystem failure; assign a designated worker by the interactive assistantfor system installation based on the analysis of the initial digitaltwin information, wherein the designated worker is a robot and a mobilecomputing device is embedded in the robot; display an installation mapon a display of a mobile computing device based on the initial digitaltwin information provided to the mobile computing device through theinteractive assistant; perform system installation by repeatedlyperforming operations until determining that the system failure isrepaired, the operations comprising: receiving images from an imagingdevice and text from the mobile computing device and storing the imagesas image data and the text as text data in the cloud server, wherein theimages show one or more components of the system being installed and thetext includes input from the designated worker; receiving updateddigital twin information; determining whether the failure is repairedbased on the image data, text data, and updated digital twin data; upondetermining that the failure is not repaired, determining a first actionto be performed by the designated worker based on analysis of the imagedata, text data, and the updated digital twin information, wherein thefirst action includes checking for an issue with a first component; uponreceiving a communication from the designated worker that the issue doesnot exist with the first component, recommending that the designatedworker check for the issue with a second component; upon receiving acommunication from the designated worker that the issue exists with thesecond component, determining a second action to be performed by thedesignated worker based on analysis of the image data, text data, andthe initial digital twin information; and recommend the second action tothe designated worker via communication to the mobile computing devicefrom the cloud server; display updated map on the display of the mobilecomputing device based on the updated twin information; and displayinstallation is completed on the display of the mobile computing deviceafter the system is installed based on analysis of the interactiveassistant.
 9. The non-transitory computer-readable medium storingsoftware of claim 8, wherein determining the second action includes:converting the image data, text data, and updated digital twin data intoa structured form, extracting features from the structured form, andapplying weights to extracted features to create weighted features;applying an action predictor including deep neural network layers to theweighted features to generate the second action.
 10. The non-transitorycomputer-readable medium storing software of claim 8, wherein theinitial digital twin information comprises a virtual representation ofthe system installed in a building or a virtual representation of acomponent for the system.
 11. The non-transitory computer-readablemedium storing software of claim 8, wherein the issue is an oil leak.12. The non-transitory computer-readable medium storing software ofclaim 8, further comprising recommending a tool for the designatedworker to use based on analysis of the image data, text data, and theinitial digital twin information.
 13. The non-transitorycomputer-readable medium storing software of claim 8, wherein theoperation of recommending the second action to be performed includesoutputting the recommended second action to the designated worker byusing a speaker of the mobile computing device.
 14. The non-transitorycomputer-readable medium storing software of claim 8, wherein the textindicates information related to a status of the system being installed.15. An interactive assistant for installing a system and for dynamicallyrecommending one or more actions for installing the system, theinteractive assistant comprising: one or more computers and anon-transitory computer readable medium storing instructions that areexecutable by the one or more computers to: access initial digital twininformation from a digital twin stored on a cloud server through aninteractive assistant stored on the cloud server for systeminstallation; analyze the initial digital twin information to detect asystem failure; assign a designated worker by the interactive assistantfor system installation based on the analysis of the initial digitaltwin information, wherein the designated worker is a robot and a mobilecomputing device is embedded in the robot; display an installation mapon a display of a mobile computing device based on initial digital twininformation provided to the mobile computing device through theinteractive assistant; perform system installation by repeatedlyperforming operations until determining that the system failure isrepaired, the operations comprising: receiving images from an imagingdevice and text from the mobile computing device and storing the imagesas image data and the text as text data in the cloud server, wherein theimages show one or more components of the system being installed and thetext includes input from the designated worker; receiving updateddigital twin information; determining whether the failure is repairedbased on the image data, text data, and updated digital twin data; upondetermining that the failure is not repaired, determining a first actionto be performed by the designated worker based on analysis of the imagedata, text data, and the updated digital twin information, wherein thefirst action includes checking for an issue with a first component; uponreceiving a communication from the designated worker that the issue doesnot exist with the first component, recommending that the designatedworker check for the issue with a second component; upon receiving acommunication from the designated worker that the issue exists with thesecond component, determining a second action to be performed by thedesignated worker based on analysis of the image data, text data, andthe initial digital twin information; and recommend the second action tothe designated worker via communication to the mobile computing devicefrom the cloud server; display updated map on the display of the mobilecomputing device based on the updated twin information; and displayinstallation is completed on the display of the mobile computing deviceafter the system is installed based on analysis of the interactiveassistant.
 16. The interactive assistant of claim 15, whereindetermining the second action includes: converting the image data, textdata, and updated digital twin data into a structured form, extractingfeatures from the structured form, and applying weights to extractedfeatures to create weighted features; applying an action predictorincluding deep neural network layers to the weighted features togenerate the first action.
 17. The interactive assistant of claim 15,wherein the initial digital twin information comprises a virtualrepresentation of the system installed in a building ora virtualrepresentation of a component for the system.